Automatic eye tracking control

ABSTRACT

An apparatus and method are provided for rapid and hands free switching among plural devices. Signals from such devices include output from plural security cameras, outputs from plural command and control devices, outputs from plural sensors in, e.g., a chemical plant or nuclear power plant, etc. An eye tracker observes changes in an operator&#39;s line of sight, thereby inferring a change in the particular one of the plural devices the observer now wishes to observe, and, through an interlock, switches control of that device to the operator. This eliminates the need for the operator to manually switch control, e.g. by keyboard strokes, mouse clicks, or by manually causing lengthy dwell time on the selected device. This in turn makes switching faster, and reduces operator fatigue by considerably reducing physical actions necessary to switch among devices.

STATEMENT OF GOVERNMENT INTEREST

The invention described was made in the performance of official dutiesby one or more employees of the Department of the Navy, and thus, theinvention herein may be manufactured, used or licensed by or for theGovernment of the United States of America for governmental purposeswithout the payment of any royalties thereon or therefor.

BACKGROUND

The invention pertains to systems and methods by which an operator maymonitor and control several different devices by permitting selectableswitching among the devices, the devices either standing alone or onother platforms such as military ships or aircraft.

Conventional systems in which an operator monitors in real time severaldifferent locations are widespread, in law enforcement, the military,and in the larger society, e.g. the perimeter about a chemical ornuclear plant. A common feature of such systems is use of a plurality ofmonitors, such as plural security cameras or the like, the output ofeach being viewed by an operator, who can selectably switch controlamong the monitors so as to operate a particular selected one, e.g. topan/tilt/zoom a selected camera.

SUMMARY

Conventional tracking systems yield disadvantages addressed by variousexemplary embodiments of the present invention. Accordingly, objectivesachievable by exemplary embodiments include reduction of the effort,physical and mental, to operate such systems, increasing the number ofdevices in such systems that an operator can control effectively at onetime, decreasing the time an operator needs to switch among devices,reducing operator stress caused by frequent shifting among systemsmonitors, and permitting an operator to switch control among monitors insuch systems without the need to use hands, or any operator extremity.

In accordance with these and other objects made apparent hereinafter,the invention concerns a system having a plurality of monitors, adisplay apparatus, and an eye tracker. Each of the monitors is adaptedto perform one or more functions controllable by an operator, and inwhich the plurality of monitors is further disposed to produce an outputsignal.

The display apparatus presents, responsive to outputs of the monitors, aplurality of displays, each of the displays corresponding to arespective one of the monitors, and being visually intelligible to theoperator. The eye tracker is disposed effective to determine thespecific one of the plurality of displays the operator is looking at,and, in response, to generate a signal to enable operator control of themonitor corresponding to that display. In this manner, control among themonitors switches automatically as the operator switches the displaylooked at, and thus spares the operator the need to manually send asignal change enable or disable monitors.

These and other objects are further understood from the followingdetailed description of particular embodiments of the invention.However, the invention is capable of extended application beyond theprecise details of these embodiments. Changes and modifications can bemade to the embodiments that do not affect the spirit of the invention,nor exceed its scope, as expressed in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

These and various other features and aspects of various exemplaryembodiments will be readily understood with reference to the followingdetailed description taken in conjunction with the accompanyingdrawings, in which like or similar numbers are used throughout, and inwhich:

FIG. 1 is an isometric elevational view of an exemplary embodiment,shown with an operator therefor;

FIG. 2 is a schematic view of the embodiment; and

FIG. 3 is an elevation view of an alternate embodiment.

DETAILED DESCRIPTION

In the following detailed description of exemplary embodiments of theinvention, reference is made to the accompanying drawings that form apart hereof, and in which is shown by way of illustration specificexemplary embodiments in which the invention may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention. Other embodiments may be utilized,and logical, mechanical, and other changes may be made without departingfrom the spirit or scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined only by the appendedclaims.

In accordance with a presently preferred embodiment of the presentinvention, the components, process steps, and/or data structures may beimplemented using various types of operating systems, computingplatforms, computer programs, and/or general purpose machines. Inaddition, those of ordinary skill in the art will readily recognize thatdevices of a less general purpose nature, such as hardwired devices, orthe like, may also be used without departing from the scope and spiritof the inventive concepts disclosed herewith. General purpose machinesinclude devices that execute instruction code. A hardwired device mayconstitute an application specific integrated circuit (ASIC) or afloating point gate array (FPGA) or other related component.

Conventional systems require that the operator view outputs of thecameras and, depending on what the operator sees, switch to oneparticular camera and control it, for example to pan/tilt/zoom a camerato get a better look as something suspicious.

The first such example includes the Department of Homeland Security'sVirtual-Fence, a series of networked microwave towers to hold sensors,cameras and communications equipment, helping agents monitor the borderto detect movement and keep out illegal immigrants and drug smugglers.Because the system has plural cameras and other sensors, to operate thesystem effectively the system operator must constantly manually shift acontrol to shift among the sensors. This results in the operatorconstantly executing numerous physical movements that require acorresponding mental decision at each step, a process which can quicklyoverload the operator mentally, and quickly wear the operator outphysically. As a practical matter, this limits how long an operator caneffectively work the system at one time, and limits how many devices anagent can effectively manage at one time. A particular difficulty withthis way of operating the system is that it requires the operator toperform multiple steps when switching between devices, for example werethe operator about to switch from one device to another, the operatorwould, first, have to identify which device to switch to, then, second,to execute the device switch, and, third, to operate the device.

The second such example involves Camera surveillance systems in storesthat permit a viewer to monitor plural critical points in the storesimultaneously so as to keep an eye on inventory and prevent its theft.The constant executing of switches among cameras limits both effectiveoperator time on the system, and the number of cameras the operator canmanage.

The third such example can be Shipboard Surveillance Systems such as areon U.S. Navy ships, in which an operator must simultaneously monitoroutputs from plural cameras and other detectors to guard against threatsto the ship. Information given to the inventors from experience on oneNavy warship was that, with more than two contacts to monitor, a systemoperator would be saturated.

The fourth such example involves Security systems for the perimeter ofsecure facilities, such as chemical or nuclear plants or military bases.Studies have shown that in the Virtual Perimeter Monitoring System atthe Naval Surface Warfare Center, Dahlgren Division, the operator mustexecute a minimum of five physical steps to control one camera, and thusmust constantly be executing those five steps again and again.

The sixth example represents a single operator monitoring multipleunmanned vehicle (UV) platforms, who needs to quickly alter the movementof a single UV. To gain control of a single UV, the operator mustperform several steps to control the platform, or assets on theplatforms, which increase the time it takes for an operator to gaincontrol of the UV.

With reference to the drawing figures, wherein like numbers indicatelike parts throughout the several views, FIGS. 1 and 2 show anembodiment according to the invention in which an operator 19 canselectably view and control several security cameras 42 (illustrated inFIG. 2 as four in number, although the exact number as shown isarbitrary and not limiting).

FIG. 1 in particular shows an operator 19 sitting at a work stationhaving, inter alia, a computer screen 10 divided by dotted lines 13 intoquadrants 10′, 10″, 10′″, and 10″″. These dotted lines are for purposesof illustration, and in practice need not appear on screen 10, moreabout which below. Screen 10 sits, via stand 12 and base 14, on tabletop 16, which has on it keyboard 24, and mouse 26, connected byrespective cables 25 and 27 to computer 30.

Computer 30 communicates with screen 10 by cable 11, as well as withmember 18 by cable 19, and with other elements (see FIG. 2) by cable 32.Operator 19 is seated facing screen 10, and looks directly at a pointwithin quadrant 10′″ along line of sight 22. Attached to stand 12 is eyetracker 18, which scans operator 19, and, in particular, the eye ofoperator 19 along line of sight 20 to determine the direction of line ofsight 22. Given the position of operator 19 at table 16, this infers thequadrant of screen 10 which operator 10 is viewing; here, again, 10′″.

Member 18 can be any conventional eye tracker, and preferably is onethat infers line of sight 20 by measuring the ratio of the light anddark portion of operator 10's eyes as visible along line 20, because ofthe reliability of such eye trackers, although any known tracker cansuffice.

With particular reference to FIG. 2, the visual output of four cameras,42′, 42″, 42′″, 42″″ are fed along corresponding lines 43′, 43″, 43′″,43″″ in parallel to multiplexer 40, which forwards the outputs in theform of a time division multiplexed signal, along cable 32 to computer30, which also receives input from eye tracker 18 as discussed above.Computer 30 de-multiplexes and transmits camera outputs to display 10such that output from camera 42′ displays in quadrant 10′ (FIG. 1),output from 42″ displays in quadrant 10″, output from 42′″ displays inquadrant 10′″, and output from 42″″ displays in quadrant 10″″, etc.

Directing the outputs from plural cameras 42 to the correct quadrant ofdisplay 10 is done preferably by programming computer 30 to do so,although any conventional scheme to do so will suffice. One could, ofcourse, use four different displays, with computer 30 sending respectivede-multiplexed outputs to each, but the use of one display such asmember 10 in FIG. 1 requires less hardware, and is likely more easilyviewable by operator 19. For purposes of this disclosure, the termdisplay includes any array of the outputs of plural monitors that anoperator can selectably view at the same time, whether or not all viewsreside in a unitary display device.

Computer 30 communicates along line 35 to switch 36, which communicatesvia line 37 with cameras 42 in parallel. Responsive to input of eyetracker 18, computer 30 determines which of quadrants 10 operator 19 isviewing, and enables camera control signals from keyboard 24 and mouse26 to pass through to the particular camera 42 associated with thatquadrant. One preferably does this by providing each camera 42′, 42″,42″″, 42″″ with a digital interlock (not shown) by which computer 30 viaswitch 36 can provide a digital code by which to enable a selected oneof cameras 42, and disable the remaining cameras.

Thus enabled, lines 35, 37 can communicate commands from keyboard 24 andmouse 26 to the enabled camera, for example the common commands to pan,tilt, zoom, etc. Upon shifting operator line of sight 22 (FIG. 1) to adifferent quadrant of display 10, eye tracker 18 detects the change andsignals the change to computer 30, which in turn signals switch 36 todisable the operative one of cameras 42, and enable for operator controlthe camera associated with the new quadrant. In this manner, the handsof operator 19 can remain on mouse 26 and the critical keys of keyboard24 that control camera operation, without having additionally to makephysical actions to move among the views of the four cameras 42. Beforeswitching from one camera 42 to another, the system must wait anappropriate time to eliminate spurious switching due to inadvertentshort duration operator movements, such as eye blinking or fidgeting inone's chair, etc.

Although FIG. 2 shows communication between computer 30 and cameras 42by way of a physical switch 36, this is for purposes of illustration.The switching function above described can advantageously be doneinternally to computer 30 by appropriate conventional programming, andcause direction of control signals to and from cameras 42 viamultiplexer 40.

FIG. 3 shows an alternative scheme in which eye tracker 18′ mountsdirectly on operator 19′, here on head covering 50, for example abaseball cap, industrial hard hat, head band, etc. Eye tracker 18′directly scans operator 19's line of sight 22′ as in the scheme of FIG.1. The embodiment of FIG. 3 is particularly useful in confined orcluttered spaces in which direct mounting of an eye tracker distant froman operator would likely block the eye tracker's view of the operator.

While certain features of the embodiments of the invention have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the embodiments.

What is claimed is:
 1. A system, said system usable by a human operator,comprising: a plurality of devices, each of said plurality of devicesbeing adapted to perform one or more functions controllable by saidoperator, said each of said plurality of devices being further disposedeffective to produce an output signal; a display apparatus, said displayapparatus disposed effective to present, responsive to said outputsignal of said each of said plurality of devices, a correspondingplurality of displays, each of said corresponding plurality of displayscorresponding to a respective one of said each of said plurality ofdevices, said each of said corresponding plurality of displays beingvisually intelligible to said operator; and an eye tracker, said eyetracker being disposed effective to make a determination of the specificone of said corresponding plurality of displays said operator is lookingat, said eye tracker being further adapted to generate a signaleffective to enable said operator to selectably control at least one ofsaid one or more functions of said respective one of said plurality ofdevices corresponding to said specific one of said plurality ofdisplays.
 2. The system of claim 1, wherein said eye tracker is mountedon said display apparatus.
 3. The system of claim 1, wherein said eyetracker is mounted on the head of said operator.
 4. The system of claim1, wherein said display is one screen subdivided into a plurality ofportions, each of said plurality of portions corresponding to arespective one of said each of said plurality of displays.
 5. The systemof claim 1, wherein said display is a plurality of separate displays. 6.The system of claim 1, wherein said plurality of devices is a pluralityof cameras, and said each of said plurality of displays presents thevisual image produced by a corresponding one of said plurality ofcameras.
 7. The system of claim 6, wherein said one or more functions isa member of the group consisting of panning, tilting, or zooming saidspecific one of said plurality cameras.